Rewards Analytics and Distribution Dashboard for Quantification Review¶

This goal of this document is to offer an easy way to process the outputs of the praise reward system and perform an analysis of the resulting token reward distribution. It should be considered a work-in-progress.

Out[6]:

Distribution report for round-1

  • This period covers praise given between 2021-07-11 and 2021-07-30.
  • We allocated a total of 2546 TEC tokens for rewards.
  • Duplicate praise received a weighting of 0.1 the value of the original praise.
  • We assigned 3 quantifiers per praise instance.
  • Praise receiver names were not hidden behind pseudonyms during quantification

Praise Data Visualization¶

Rating distribution¶

Since praise gets valued on a scale, we can take a look at how often each value of the scale gets assigned by quantifiers. Note: how to process the duplicate scores? For now, just delete them.

Praise Reward Distribution¶

We can now take a look at the distribution of the received praise rewards. You can toggle the inclusion of the different sources by clicking on the legend.

Praise Giving Distribution¶

We can also take a look at the distribution of the people giving praise.

Praise Flows¶

Now for something more fun: let's surface the top "praise flows" from the data. Thanks to @inventandchill for this awesome visualization! On one side we have the top 20 praise givers separately (modifiable by changing the variable n_senders), on the other the top 25 receivers (modifiable by changing the variable n_receivers). The people outside the selection get aggregated into the "REST FROM" and "REST TO" categories.

Out[13]:

Praise Reward Reception Change History¶

See if we have more people giving and receiving praise now, as well as the total rating. TO DO

Quantifier Data¶

Let's take a closer look at the quantification process and and see if we can spot any problems:

Praise Outliers¶

To aid the revision process, we highlight disagreements between quantifiers.

Outliers sort by spreads¶

Here we generate a table which sorts the praise by the size of the spread between the highest and lowest scores. It gives us an overview of the spread distribution.

For an exhaustive list, take a look at the exported file "praise_outliers.csv" .

This is a visual aid. ATTENTION! If there are several praise instances with similar spread and quant score, all but one end up "hidden" on the chart.

Praise score by quantifier -- outliers among the quantifiers?¶

Let's see the range of praise scores every quantifier gave to see the behavior difference of quantifiers.

To interpret the box plot:

  • Bottom horizontal line of box plot is minimum value

  • First horizontal line of rectangle shape of box plot is First quartile or 25%

  • Second horizontal line of rectangle shape of box plot is Second quartile or 50% or median.

  • Third horizontal line of rectangle shape of box plot is third quartile or 75%

  • Top horizontal line of rectangle shape of box plot is maximum value.

Agreement on duplication¶

Among 597 praises, 133 (22.20%) do not agree on duplication

Praise instances with disagreements in duplication are collected in 'results/duplication_examination.csv'. To compare, look at the last 4 columns: 'DUPLICATE MSG 1/2/3' and 'ORIGINAL MSG'.

Agreement on dismissal¶

Among 597 praises, 15 (2.50%) do not agree on dismissal

Praise instances with disagreements in dismissal are collected in'results/dismissal_disaggreed.csv'. You can further look into who dismissed and who did not.